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Abstract 

A general method of coding over expansion is proposed, which allows one to reduce the highly 
non-trivial problems of coding over analog channels and compressing analog sources to a set of 
much simpler subproblems, coding over discrete channels and compressing discrete sources. More 
specifically, the focus of this paper is on the additive exponential noise (AEN) channel, and lossy 
compression of exponential sources. Taking advantage of the essential decomposable property of these 
channels (sources), the proposed expansion method allows for mapping of these problems to coding over 
parallel channels (respectively, sources), where each level is modeled as an independent coding problem 
over discrete alphabets. Any feasible solution to the resulting optimization problem after expansion 
corresponds to an achievable scheme of the original problem. Utilizing this mapping, even for the cases 
where the optimal solutions are difficult to characterize, it is shown that the expansion coding scheme 
still performs well with appropriate choices of parameters. More specifically, theoretical analysis and 
numerical results reveal that expansion coding achieves the capacity of AEN channel in the high SNR 
regime. It is also shown that for lossy compression, the achievable rate distortion pair by expansion 
coding approaches to the Shannon limit in the low distortion region. Remarkably, by using capacity- 
achieving codes with low encoding and decoding complexity that are originally designed for discrete 
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alphabets, for instance polar codes, the proposed expansion coding scheme allows for designing low- 
complexity analog channel and source codes. 


I. Introduction 

A. Background 

The field of ehannel coding was started with Shannon’s famous theorem proposed in 1948 
||T|, which shows that the channel capacity upper bounds the amount of information that can be 
reliably transmitted over a noisy communication channel. After this result, seeking for practical 
coding schemes that could approach channel capacity became a central objective for researchers. 
On the way from theory to practice, many coding schemes are proposed. Different types of codes 
emerge in improving the performance, giving consideration to the trade-off between coding 
complexity and error decay rate. 

The history of channel coding traces back to the era of algebraic coding, including the well- 
known Hamming codes Q, Golay codes Q, Reed-Muller codes 0 0 . Reed-Solomon codes 
Il6|, lattice codes 0, and others 0. However, despite enabling significant advances in code 
design and construction, algebraic coding did not turn out to be the most promising means 
to approach the Shannon limit. The next era of probabilistic coding considered approaches 
that involved optimizing code performance as a function of coding complexity. This line of 
development included convolutional codes 0, and concatenated codes [10| at earlier times, 
as well as turbo codes pT| and low-density parity-check (LDPC) codes [12| [13| afterwards. 
Recently, polar codes [fT4| have been proved to achieve Shannon limit of binary-input symmetric 


channels with low encoding and decoding complexity. In another recent study [I5| [I6|, new 
types of rateless codes, viz. spinal codes, are proposed to achieve the channel capacity. 

Another well-studied (and practically valuable) research direction in information theory is 
the problem of compression of continuous-valued sources. Given the increased importance of 
voice, video and other multimedia, all of which are typically “analog” in nature, the value 
associated with low-complexity algorithms to compress continuous-valued data is likely to remain 
significant in the years to come. For discrete-valued “finite-alphabet” problems, the associated 
coding theorem and practically-meaningful coding schemes are well known. Trellis based 


quantizers [18| are the first to achieve the rate distortion trade-off, but with encoding complexity 


scaling exponentially with the constraint length. Later, Matsunaga and Yamamoto [ 191 show that a 
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low density parity check (LDPC) ensemble, under suitable conditions on the ensemble structure, 
can achieve the rate distortion bound using an optimal decoder. p0[ shows that low density 
generator matrix (LDGM) codes, as the dual of LDPC codes, with suitably irregular degree 
distributions, empirically perform close to the Shannon rate-distortion bound with message¬ 
passing algorithms. More recently, polar codes are shown to be the first provably rate distortion 


limit achieving codes with low complexity [211. In the case of analog sources, although both 
practical coding schemes as well as theoretical analysis are very heavily studied, a very limited 
literature exists that connects the theory with low-complexity codes. The most relevant literature 


in this context is on lattice compression and its low-density constructions [22|. Yet, this literature 
is also limited in scope and application. 


B. Contributions and Organization 

The problem of coding over analog noise channels is highly non-trivial in general. To this 
end, a method of modulation is commonly utilized to map discrete inputs to analog signals for 
transmission through the physical channel In this paper, we focus on designing and coding 
over such mappings. In particular, we propose a new coding scheme for general analog channels 
with moderate coding complexity based on an expansion technique, where channel noise is 
perfectly or approximately represented by a set of independent discrete random variables (see 
Fig. [g. Via this representation, the problem of coding over an analog noise channel is reduced 
to that of coding over parallel discrete channels. We focus on additive exponential noise (AEN), 
and we show that the Shannon limit, i.e., the capacity, is achievable for AEN channel in the 
high SNR regime. More precisely, for any given e < 1, it is shown that the gap to capacity is at 
most 5e log(e) when at least —2 log(e) -f log(SNR) number of levels are utilized in the coding 
scheme together with embedded binary codes. Generalizing results to g-ary alphabets, we show 
that this gap can be reduced more. The main advantage of the proposed method lies on its 
complexity inheritance property, where the encoding and decoding complexity of the proposed 
schemes follow that of the embedded capacity achieving codes designed for discrete channels, 
such as polar codes and spinal codes. 

In the second part of this paper, we present an expansion coding scheme for compressing 
of analog sources. This is a dual problem to the channel coding case, and we utilize a similar 
approach where we consider expanding exponential sources into binary sequences, and coding 
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Expansion Channel Coding 





Fig. 1. Illustration of expansion channel coding framework. In this example, an analog noise channel is expanded 
into a set of discrete channels with index from —Li to L 2 . Channel noise 2 is considered as its binary expansion 
2 = • ■ • 2^2 • • ■ 2 i 20 . 2-1 • • • Z-Li • ■ • , and similar expansions are adopted to channel input and output. Carries exist between 
neighboring levels. 


over the resulting set of parallel diserete sourees. We show that this seheme’s performanee 
ean get very elose to the rate distortion limit in the low distortion regime (i.e., the regime of 
interest in praetiee). More preeisely, the gap between the rate of the proposed seheme and the 
theoretieal limit is shown to be within a eonstant gap (51og(e)) for any distortion level D when 
at least — 21og(A^Zi)) number of levels are utilized in the eoding seheme (where, 1/A is the 
mean of the underlying exponential souree). Moreover, this expansion eoding seheme ean be 
generalized to Laplaeian sourees (two-sided symmetrie exponential distribution), where the sign 
bit is eonsidered separately and eneoded perfeetly to overeome the diffieulty of souree value 
being negative. 

The rest of paper is organized as follows. Related work is investigated and summarized 
in Seetion The expansion eoding seheme for ehannel eoding is detailed and evaluated in 
Seetion |I^ The expansion souree eoding framework and its applieation to exponential sourees 
are demonstrated in Seetion IV Finally, the paper is eoneluded in Seetion |Vj 
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Fig. 2. Illustration of multilevel coding framework. In this example, multilevel coding scheme is illustrated. Compared to 
expansion coding in Fig. [T] only channel input is expressed by multi-levels, but not the channel noise. 


II. Related Work 

Multilevel coding is a general coding method designed for analog noise channels with a 


flavor of expansion [24|. In particular, a lattice partition chain Ai/ ■ ■ ■ /A^-i/A^ is utilized to 
represent the channel input, and, together with a shaping technique, the reconstructed codeword is 
transmitted to the channel. It has been shown that optimal lattices achieving Shannon limit exist. 
However, the encoding and decoding complexity for such codes is high, in general. In the sense 
of representing the channel input, our scheme is coincident with multilevel coding by choosing 
Ai = Ar = for some Li, L 2 G Z’*', where coding of each level is over g-ary 

finite field (see Fig. [^. The difference in the proposed method is that besides representing the 
channel input in this way, we also “expand” the channel noise, such that the coding problem for 
each level is more suitable to solve by adopting existing discrete coding schemes with moderate 
coding complexity. Moreover, by adapting the underlying codes to channel-dependent variables, 
such as carries, the Shannon limit is shown to be achievable by expansion coding with moderate 
number of expanded levels. 


The deterministic model, proposed in 1251, is another framework to study analog noise channel 
coding problems, where the basic idea is to construct an approximate channel for which the 
transmitted signals are assumed to be noiseless above a certain noise level. This approach has 
proved to be very effective in analyzing the capacity of networks. In particular, it has been 
shown that this framework perfectly characterizes degrees of freedom of point-to-point AWGN 
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Fig. 3. Comparison of noise models between expansion coding and deterministic model. The noise models of each level 
for expansion coding and deterministic model are illustrated. Deterministic model cut the noise to a certain level, and expansion 
coding has a smooth transaction regime. 


channels, as well as some multi-user channels. In this sense, our expansion coding scheme can 
be seen as a generalization of these deterministic approaches. Here, the effective noise in the 
channel is carefully calculated and the system takes advantage of coding over the noisy levels 
at any SNR. This generalized channel approximation approach can be useful in reducing the 
large gaps reported in the previous works, because the noise approximation in our work is much 
closer to the actual distribution as compared to that of the deterministic model (see Fig. [^. 

There have been many attempts to utilize discrete codes for analog channels (beyond simple 
modulation methods). For example, after the introduction of polar codes, considerable attention 
has been directed towards utilizing their low complexity property for analog channel coding. 
A very straightforward approach is to use the central limit theorem, which says that certain 
combinations of i.i.d. discrete random variables converge to a Gaussian distribution. As reported 


in [161 and [26|, the capacity of AWGN channel can be achieved by coding over large number 
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of BSCs, however, the eonvergenee rate is linear whieh limits its application in practice. To this 


end, [271 proposes a MAC based scheme to improve the convergence rate to exponential, at the 


expense of having a much larger field size. A newly published result in [1^| attempts to combine 
polar codes with multilevel coding, however many aspects of this optimization of polar-coded 
modulation still remain open. Along the direction of this research, we also try to utilize capacity 
achieving discrete codes to approximately achieve the capacity of analog channels. 

The additive exponential noise (AEN) channel is of particular interest as it models worst-case 
noise given a mean and a non-negativity constraint on noise [ |29| . In addition, the AEN model 
naturally arises in non-coherent communication settings, and in optical communication scenarios. 


(We refer to |29| and [30| for an extensive discussion on the AEN channel.) Verdu derived the 


optimal input distribution and the capacity of the AEN channel in [29|. Martinez, on the other 
hand, proposed the pulse energy modulation scheme, which can be seen as a generalization of 
amplitude modulation for the Gaussian channels. In this scheme, the constellation symbols are 
chosen as c{i — 1)^ for i = 1, • • ■ ,2^ with a constant c, and it is shown that the information 
rates obtained from this constellation can achieve an energy (SNR) loss of 0.76 dB (with the 
best choice of I = ^(1 + V^)) compared to the capacity in the high SNR regime. Another 
constellation technique for this coded modulation approach is recently considered in pT| , where 
log constellations are designed such that the real line is divided into (2M — 1) equally probable 
intervals. M of the centroids of these intervals are chosen as constellation points, and, by a 
numerical computation of the mutual information, it is shown that these constellations can achieve 
within a 0.12 dB SNR gap in the high SNR regime. Our approach, which achieves arbitrarily close 
to the capacity of the channel, outperforms these previously proposed modulation techniques. 

In the domains of image compression and speech coding, Eaplacian and exponential distribu¬ 
tions are widely adopted as natural models of correlation between pixels and amplitude of voice 
[ |32| . Exponential distribution is also fundamental in characterizing continuous-time Markov 
processes [ [29| . Although the rate distortion functions for both have been known for decades, there 
is still a gap between theory and existing low-complexity coding schemes. The proposed schemes, 
primarily for the medium to high distortion regime, include the classical scalar and vector 


quantization schemes [33 j, and Markov chain Monte Carlo (MCMC) based approach in [34|. 
However, the understanding of low-complexity coding schemes, especially for the low-distortion 
regime, remains limited. To this end, our expansion source coding scheme aims to approach the 









rate distortion limit with practical encoding and decoding complexity. By expanding the sources 
into independent levels, and using the decomposition property of exponential distribution, the 
problem has been remarkably reduced to a set of simpler subproblems, compression for discrete 
sources. 


III. Expansion Channel Coding 

A. Intuition 

In general, expansion channel coding is a scheme of reducing the problem of coding over an 
analog channel to coding over a set of discrete channels. In particular, we consider the additive 
noise channel given by 

Yi = Xj + Zi, i = 1, • • • ,n, (1) 


where Xj are channel inputs with alphabet X (possibly having channel input requirements, such 
as certain moment constraints); Y* are channel outputs; Zj are additive noises independently 
and identically distributed with a continuous probability density function; n is block length. 
We represent the inputs as Xi.„ = (Xi, • ■ ■ ,X„}. (Similar notation is used for other variables 
throughout the sequel.) 

When communicating, the transmitter conveys one of the messages, M, which is uniformly 
distributed in Ad = {1, • ■ ■ ,2"^^}; and it does so by mapping the message to the channel 
input using encoding function 0(-) ; Ad —)■ X'^ such that Xi;„(M) = 0(M). The decoder uses 
the decoding function ^(■) to map its channel observations to an estimate of the message. 
Specifically, ip{-) : 3^™ —)■ Ad, where the estimate is denoted by M = t/’(Yi.„). A rate R is said 
to be achievable, if the average probability of error defined by 


P = 

^ e 


\M\ 


Pr{M M| M is sent.} 


MeM 


can be made arbitrarily small for large n. The capacity of this channel is denoted by C, which 


is the maximum achievable rate R, and its corresponding optimal input distribution is denoted 
as 

Our proposed coding scheme is based on the idea that by “expanding” the channel noise (i.e.. 


representing it by its g-ary expansion), an approximate channel can be constructed, and proper 
coding schemes can be adopted to each level in this representation. If the approximation is close 
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enough, then the coding schemes that are optimal for each level can be translated to an effective 
one for the original channel. More formally, consider the original noise Z and its approximation 
Z, which is defined by the truncated g-ary expansion of Z. For this moment, we simply take 
g = 2 (i.e., considering binary expansion), and leave the general case for later discussion. 

L2 

Z ^ Z^'S" 

l=-Li 

where Z®'^" represents the sign of Z, taking a value from { —, +}; Z;’s are mutually independent 
Bernoulli random variables. By similarly expanding the channel input, we convert the problem 
of coding over analog channels to coding over a set of binary discrete channels. This mapping 
is highly advantageous, as capacity achieving discrete codes can be adopted for coding over 
the constructed binary channels. Assume the input distributions for sign channel and discrete 
channel at I are represented by and X; correspondingly, then an achievable rate (via random 
coding) for the approximated channel is given by 

^^/(X;X + Z), 

where 

L2 

X ^ X^'S" 2'Xi. 

l=-Li 

By adopting the same coding scheme over the original channel, one can achieve a rate given by 

R = I{X-X + Z). 

The following result provides a theoretical basis for expansion coding. (Here, A denotes con¬ 
vergence in distribution.) 

Theorem 1. 7/' Z A Z and X A X*, as Ti, L 2 — )■ 00 , where X* ~ ^he optimal input 

distribution for the original channel, then R ^ C. 

The proof of this theorem follows from the continuity property of mutual information. In 
other words, if the approximate channel is close to the original one, and the distribution of the 
input is close to the optimal input distribution, then the expansion coding scheme will achieve 
the capacity of the channel under consideration. 
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B. AEN Channel Coding Problem Setup 


In this section, we consider an example where expansion channel coding can achieve the 
capacity of the target channel. The particular channel considered is an additive exponential noise 
(AEN) channel, where the channel noise Zj in ([T]) is independently and identically distributed 
according to an exponential density with mean Ez, i.e., omitting the index i, noise has the 
following density: 

fz{z) = ^e~^u{z), (2) 

Ez 

where u{z) = 1 for z > 0 and u{z) = 0 otherwise. Moreover, channel input Xj in ([T]) is restricted 
to be non-negative and satisfies the mean constraint 


1 

- Ve[X,] <Ex. 

n 


2=1 


The capacity of AEN channel is given by [29|, 


(3) 


Caen — log(l + SNR), 


(4) 


where SNR = Ex/Ez, and the capacity achieving input distribution is given by 

Ex _ =2 _ . . Ez 


fxix) = 


-e^x+^zu{x) + 




(5) 


{Ex + EzY '' ' Ex + Ez 

where 6{x) = 1 if and only if a: = 0. Here, the optimal input distribution is not exponentially 
distributed, but a mixture of an exponential distribution with a delta function. However, we 
observe that in the high SNR regime, the optimal distribution gets closer to an exponential 
distribution with mean Ex, since the weight of delta function approaches to 0 as SNR tends to 
infinity. 


C. Binary Expansion of Exponential Distribution 

The basis of the proposed coding scheme is the expansion of analog random variables to dis¬ 
crete ones, and the exponential distribution emerges as a first candidate due to its decomposition 
property. We show the following lemma, which allows us to have independent Bernoulli random 
variables in the binary expansion of an exponential random variable. 
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Lemma 2. Let be independent Bernoulli random variables with parameters given by bi, 
i.e., Pr{Bi = 1} = bi, and consider the random variable defined by 

OO 

B = ^ 2%. 

l=—oo 

Then, the random variable B is exponentially distributed with mean i.e., its pdf is given by 

/B(6) = Ae-"^ 6>0, 

if and only if the choice of bi is given by 

“ 1 + • 

Proof: See Appendix ■ 

This lemma reveals that one ean reeonstruet exponential random variable from a set of inde¬ 
pendent Bernoulli random variables perfeetly. Fig. [^illustrates that the distribution of reeovered 
random variable from expanded levels (obtained from the statisties of 100, 000 independent 
samples) is a good approximation of original exponential distribution. 

A set of typieal numerieal values of biS for A = 1 is shown in Fig. [^ It is evident that bi 
approaehes to 0 for the “higher” levels and approaehes 0.5 for what we refer to as “lower” levels. 
Henee, the primary non-trivial levels for whieh eoding is meaningful are the so-ealled “middle” 
ones, whieh provides the basis for truneating the number of levels to a finite value without a 
signifieant loss in performanee. 

D. Expansion Coding for AEN Channel 

We eonsider the binary expansion of the ehannel noise 

L2 

( 6 ) 

l=-Li 

where Zj ^ are i.i.d. Bernoulli random variables with parameters 

- Pr{Zz = 1} = 1 = -L,,...,L2. (7) 

By Lemma Zj A Zj as Li,L 2 —)■ oo. In this sense, we approximate the exponentially 
distributed noise perfeetly by a set of diserete Bernoulli distributed noises. Similarly, we also 
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Fig. 4. Distribution of recovered random variable from expanded levels, comparing with original exponential distribution 

(A = 1). 100,000 samples are generated from the expansion form of discrete random variables, where expansion levels are 
truncated from —10 to 10. 

expand channel input and output as in the following, 

L2 



( 8 ) 


l=-Lx 

L2 



(9) 


l=-Li 

where Xu and ^ are also Bernoulli random variables with parameters Pr{X/ = 1} = pi 
and Pr{Y; = 1} = r; correspondingly. Here, the channel input is chosen as zero for levels 
I ^ {—Li, • • ■ ,^ 2 }. Noting that the summation in the original channel is a sum over real 
numbers, we do not have a binary symmetry channel (BSC) at each level (from X/s to Y^s). If 
we could replace the real sum by modulo-2 sum such that at each level I we have an independent 
coding problem, then any capacity achieving BSC code can be utilized over this channel. (Here, 
instead of directly using the capacity achieving input distribution of each level, we can use 
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Fig. 5. Numerical results for a set of bi with A = 1. X-axis is the level index for binary expansion (e.g., value —2 means 
the weight of corresponding level is 2“^), and Y-axis shows the corresponding probability of taking value 1 at each level, i.e., 
bi. 


its combination with the method of Gallager [32| to achieve a rate eorresponding to the one 
obtained by the mutual information /(X;; Y;) evaluated with an input distribution Bernoulli with 
parameter pi. This helps to approximate the optimal input distribution of the original channel.) 
However, due to the addition over real numbers, carries exist between neighboring levels, whieh 
further implies that the levels are not independent. Every level, exeept for the lowest one, is 
impaeted by carry from lower levels. In order to alleviate this issue, two schemes are proposed 
in the following to ensure independent operation of the levels. In these models of eoding over 
independent parallel channels, the total achievable rate is the summation of individual achievable 
rates over all levels. 

1) Considering carries as noise: Denoting the earry seen at level I as C* ;, which is also a 
Bernoulli random variable with parameter Pr{Ci ^ = 1} = ci, the remaining channels can be 
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represented with the following, 

where the effeetive noise, Z* j, is a Bernoulli random variable obtained by the eonvolution of the 
aetual noise and the earry, i.e., 

qi = Pr{Zi^i = 1} = g; ® Q = qi{l - Ci) + q(1 - qi). 

Here, the earry probability is given by the following reeursion relationship: 

• For level I = —Li, 

C-Li = 0 ; 

• For level I > —Li, 

Q+i = PiQii^ - Cl) + P/(l - qi)ci + (1 - Pi)qiCi + piqiCi. 

Using eapaeity aehieving eodes for BSC, e.g., polar eodes or spinal eodes, eombined with 
the Gallager’s method, expansion eoding aehieves the following rate by eonsidering earries as 
noise. 

Theorem 3. Expansion coding, considering carries as noise, achieves the rate for AEN channel 
given by 

L2 L2 

Ri= ® 

l=-Li l=-Li 

for any Li,L2 > 0, where pi G [0,0.5] is chosen to satisfy constraint Q, i.e., 

n .. n L2 L2 

- y^E[X,] = - E E 2'E[Xi,,l = E 2 'b < Bx. 

i=l i=l l=—L\ l=—Li 

2) Decoding carries: In this seheme, let us eonsider deeoding starting from the lowest level 
I = —Li. The reeeiver will obtain the eorreet X* for i = 1, • ■ ■ , n by using powerful diserete 
eoding at this level. As the reeeiver has the knowledge of Yj _L^, it ean determine the eorreet 
noise sequenee for z = 1, ■ ■ • , n. With this knowledge, the reeeiver ean direetly obtain 

eaeh Cj for z = 1, • ■ ■ , rz, whieh is the earry from level I = —Li to level I = —Li +1. This 
way, by iterating to higher levels, the reeeiver ean reeursively subtraet the impaet of earry bits. 
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Therefore, when there is no deeoding error at eaeh level, the effeetive ehannel that the receiver 
observes is given by 

ffi i 1, • • • , Tl, 

for / = — Li, • • ■ , L 2 . We remark that with this decoding strategy, the effective channels will no 
longer be a set of independent parallel channels, as decoding in one level affects the channels 
at higher levels. However, if the utilized coding method is strong enough (e.g., if the error 
probability decays to 0 exponentially with n), then decoding error due to carry bits can be made 
insignificant by increasing n for a given number of levels. We state the rate resulting from this 
approach in the following theorem. 

Theorem 4. Expansion coding, by decoding the carries, achieves the rate for AEN channel given 
by 

L2 L2 

R2= ^ R2,i= 52 [H{pi^qi)-Hiqi)], (11) 

l=-Li l=-Li 

for any Li,L2 > 0, where pi G [0,0.5] is chosen to satisfy constraint i.e., 

-I ^ n L2 L2 

- E = 2'E[X„1 = E 2'b < Bx. 

i=l i=l l=—L\ l=—Li 

Compared to the previous case, the optimization problem is simpler here as the rate expression 
is simply the sum of the rates obtained from a set of parallel channels. Optimizing for these two 
theoretical achievable rates require choosing proper values for pi. Note that, the optimization 
problems given by Theorem and are not easy to solve in general. Here, instead of searching 
for the optimal solutions directly, we utilize the information from the optimal input distribution 
of the original channel. Recall that the distribution in (|^ can be approximated by an exponential 
distribution with mean Ex at high SNR. Hence, one can simply choose pi from the binary 
expansion of the exponential distribution with mean Ex as an achievable scheme, i.e., 

pi^Pr:{Xi = l} = ^-^^^, / = -Li,...,L2. (12) 

We now show that this proposed scheme achieves the capacity of AEN channel in the high 
SNR regime for a sufficiently high number of levels. For this purpose, we first characterize the 
asymptotic behavior of entropy at each level for qi and qi correspondingly, where the later one 
is closely related to carries. 
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Lemma 5. The entropy of noise seen at level I, H{qi), is bounded by 

H{qi) < log e for I > rj, (13) 

H{qi) > 1 — 2^“’’loge for I < rj, (14) 

where p = log Ez- 

Proof: See Appendix ■ 

Lemma 6. The entropy of equivalent noise at level I, H{qi), is bounded by 

H{qi) < 6{l — ?7)2“^’''’^ loge for I > p, (15) 

H{qi) > 1 — 2^“^loge for I < p, (16) 

where p = log Ez- 

Proof: See Appendix ■ 

The intuitions behind these lemmas are given by the example seenario in Fig. whieh shows 
that the bounds on noise tails are both exponential. Now, we state the main result indieating the 
capaeity gap of expansion coding scheme over AEN channel. 

Theorem 7. For any positive constant e <1, if 

• Li > -loge - \ogEz; 

• ^2 > - log e + log Ex; 

• SNR > 1/e, where SNR = Ex/Ez, 
then, with the choice of pi as 0. 

1) considering carries as noise, the achievable rate given by satisfies 

Ri > C'aen — c, 

where c is a constant independent of SNR and e; 

2) decoding carries, the achievable rate given by (HD) satisfies 

R 2 > Caen - 5e log e. 

Proof: The proof of this theorem is based on the observation that the sequence of pi is 
a left-shifted version of qi at high SNR regime. As limited by power constraint, the number 
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Fig. 6. Signal and noise probabilities, and rates per level, pi, qi, pi ® qi, qi, pi ® qi and rates at each level are shown. In 
this example, Ex = 2^® and Ez = 2°, which further implies pi is a left-shifted version of qi by 15 levels. The coding scheme 
with Li = 5 and L 2 ~ 20 covers the significant portion of the rate obtained by using all of the parallel channels. 


of levels shifted is at most log(l + SNR), whieh further implies the rate we gain is roughly 
log(l + SNR) as well, when earries are deeoded. If eonsidering earries as noise, then there is 
apparent gap between the two version of noises, which leads to a constant gap for achievable rate. 
Fig. helps to illustrate key steps of the intuition, and a detailed proof with precise calculations 
is given in Appendix ■ 

By Lemma ^ Z Z, and combined with the argument in Theorem ^ we have R 2 ^ R ^iS 
Li, L 2 —)■ 00 . Hence, the coding scheme also works well for the original AEN channel. More 
precisely, expansion coding scheme achieves the capacity of AEN channel at high SNR region 
using moderately large number of expansion levels. 
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Fig. 7. Numerical results of achievable rates for AEN channels using expansion coding. Ri: The rate obtained by 
considering carries as noise. i? 2 : The rate obtained by decoding carry at each level. Solid lines represent adopting enough 
number of levels as indicated in Theorem |7] while dashed lines represent only adopting constant number of levels (not scaling 
with SNR). 


E. Numerical results 

We calculate the rates obtained from the two schemes above (.Ri as and i ?2 as ([TT])) with 
input probability distribution given by ( [T^ . 

Numerical results are given in Fig. It is evident from the figure (and also from the analysis 
given in Theorem that the proposed technique of decoding carries, when implemented with 
sufficiently large number of levels, achieves channel capacity at high SNR regime. 

Another point is that neither of the two schemes works well in low SNR regime, which 
mainly results from the fact that input approximation is only perfect for sufficiently high SNR. 
Nevertheless, the scheme (the rate obtained by decoding carries) performs close to optimal in 
the moderate SNR regime as well. 
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F. Generalization 

In the previous seetion, only binary expansion was eonsidered. Generalization to g-ary expan¬ 
sion with g > 2 is diseussed here. Note that this ehange does not impaet the expansion eoding 
framework, and the only difference lies in that each level after expansion should be modeled as 
a g-ary discrete memoryless channel. For this, we need to characterize the g-ary expansion of 
exponential distribution. Mathematically, the parameters of expanded levels for an exponential 
random variable B with parameter A can be calculated as follows: 


bi^s = Pr{Bi = s} 


^Pr{g^(g/c -f s) < B < q\qk -f s -f 1)} 

k=0 
oo 


Q-W{qk+s) _ ^-Xq'^iqk+s+l) 


k=0 

1 - 


1 - 6 -^ 9 '+' 

where I G {—Li, ..., L 2 } and s G {0,..., g — 1}. 

Based on this result, consider channel input and noise expansions as 


Pi^s = Pr{Xz = s} = 


1 — 


and 


qi^s = Pr{Zi = s} = 


^X — e~^ g-i'^Z-Ez 


X — 

Then, the achievable rate by decoding carries (note that in g-ary expansion case, carries are still 
Bernoulli distributed) can be expressed as 


^2 

-R2 = ^ [H{pifl-,q-l 0 qifl-,q-l) — H{qifl.,q_i)] , 

l=-Li 


(17) 


where Pxo:g-i ^nd gxo:g-i denote the distribution of expanded random variables at level I for 
input and noise respectively; ® represents for the vector convolution. 

When implemented with enough number of levels in coding, the achievable rates given by 
© achieves the capacity of AEN channel for any g > 2. More precisely, as shown in the 
numerical result in Fig. expansion coding with larger g can achieve a higher rate (although 
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Fig. 8. Numerical results for g-ary expansion. The achievable rates using g-ary expansion coding by decoding carries are 
illustrated. 


this enhancement becomes limited when q gets greater than 10). This property of the coding 
scheme can be utilized to trade-off number of levels (Li + L 2 ) and the alphabet size (q) to 
achieve a certain rate at a given SNR. 

IV. Expansion Source Coding 

A. Intuition 

Expansion source coding is a scheme of reducing the problem of compressing analog sources 
to compressing a set of discrete sources. In particular, consider an i.i.d. source Xi,X 2 ,... ,X„. 
A , n)-rate distortion code consists of an encoding function (^ : M” —)■ Ad, where Ad = 
{1,..., 2"^^}, and a decoding function ^ ; Ad —)■ M", which together map Xi:„ to an estimate 
Xi:„. Then, the rate and distortion pair (R, D) is said to be achievable if there exists a sequence 
of (2"^^, n)-rate distortion codes with lim E[(i(Xi;„, Xi:n)] < id for a given distortion measure of 

n^oo 
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interest The rate distortion funetion R{D) is the infimum of sueh rates, and by Shannon’s 

theorem [ [T7| |, we have: 

RiD)= min /(X;X), 

f{x\xy.E[d{Xl:„,Xl,„)]<D 

where the optimal eonditional distribution is given by f*{x\x). 

The expansion souree eoding seheme proposed here is based on the observation that by 
expanding the original analog souree into a set of independent diserete random variables, proper 
souree eoding sehemes eould be adopted for every expanded level. If this approximation in 
expansion is elose enough, then the overall distortion obtained from expansion eoding seheme 
is also elose to the optimal distortion. More formally, eonsider the original analog souree X and 
its approximation X given by (omitting index i) 

L2 

X ^ X^'S" 2'Xi, (18) 

l=-Li 

where X®‘®" represents the sign of X and takes values from {—,+}, and X/ is the expanded 
Bernoulli random variable at level 1. Similarly, if we expand the estimate by 

X ^ 2%, (19) 

l=-Li 

where represents the sign of X, random variable taking values from { — ,+}, and X; is 
independent Bernoulli random variable at level I after expansion. 

Here, we reduee the original problem to a set of souree eoding subproblems over levels —Li to 
L 2 . Similar to the ehannel eoding ease analyzed above, if X A X, and X A X*, as Li, L 2 —)■ 00 , 
then the aehieved rate distortion pair approximates the original one. Note that, in general, the 
deeomposition may not be suffieiently elose for most of the sourees, and the distribution for the 
estimate may not be suffieiently approximated. These situations add more distortion and result 
in a gap from the theoretieal limit. 

B. Exponential Source Coding Problem Setup 

In this seetion, a partieular lossy eompression example is introdueed to illustrate the effee- 
tiveness of expansion souree eoding. Consider an i.i.d. exponential souree sequenee Xi,... ,X„, 
i.e., omitting index i, eaeh variable has a pdf given by 


fx{x) = Xe X > 0, 
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where A ^ is the mean of X. Distortion measure of eoneem is the “one-sided error distortion” 
given by 


Otherwise, 

where ^ indieates eomparison of veetors element-wise (eaeh element should be greater than the 


d{Xi-n,Xi:n) = < *=1 

oo, 


other). This setup is equivalent to the one in [29|, where another distortion measure is eonsidered. 


Lemma 8 ( [291). The rate distortion function for an exponential source with the one-sided 


error distortion is given by 


R{D) = 


~\og{XD), 0 < < A 


0 , 


D>{. 


( 20 ) 


Moreover, the optimal conditional distribution to achieve the limit is given by 


/x|x(^l^) = 




X > X > 0. 


( 21 ) 


Proof: Proof is given in [29|, and it is based on the observation that among the ensemble of 
all probability density funetions with positive support set and mean eonstraint, exponential distri¬ 
bution maximizes the differential entropy. By designing a test ehannel from X to X, with additive 
noise distributed as exponential with parameter 1/D, both the infimum mutual information and 
optimal eonditional distribution ean be eharaeterized. Details ean be found in Appendix ■ 


C. Expansion Coding for Exponential Source 

Using Lemma we reeonstruet the exponential distribution by a set of diserete Bernoulli 
random variables. In partieular, the expansion of exponential souree over levels ranging from 
—Li to L 2 ean be expressed as 

L2 

X, = t = l,2,...,n, 

l=-L^ 

where Xi i are Bernoulli random variables with parameter 

p, Apr{X„ = l} = -^. (22) 
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This expansion perfectly approximates exponential source by letting Li,L 2 —)■ oo. Consider a 
similar expansion of the source estimate, i.e., 

^2 

Xj = ^ 2 = 1, 2,..., n, 

l=-Li 

where Xu is the resulting Bernoulli random variable with parameter pi = Pr{Xj / = !}• Utilizing 
the expansion method, the original problem of coding for a continuous source can be translated 
to a problem of coding for a set of independent binary sources. This transformation, although 
seemingly obvious, is valuable as one can utilize powerful coding schemes over discrete sources 
to achieve rate distortion limits with low complexity. In particular, we design two schemes for 
the binary source coding problem at each level. 

1) Coding with one-sided distortion: In order to guarantee x > x, we formulate each level 
as a binary source coding problem under the binary one-sided distortion constraint: do{xi, xi) = 
^{xi>xi}- Denoting the distortion at level / as di, an asymmetric test channel (Z-channel) from 
Xi to X; can be constructed, where 


Pv{Xi = l|Xz = 
Pv{Xi = 0\Xi = 


0 } 


di 

1 - Pi + d; ’ 


1 } = 0 . 


Based on this, we have pi — pi = di, and the achievable rate at a single level / is given by 

Rz,i = H{pi) -{1-PI + di)H (- -. (23) 

\l - Pi+ dij 

Due to the decomposability property as stated previously, the coding scheme considered is over 
a set of parallel discrete levels indexed by I = —Li,..., L 2 . Thus, by adopting rate distortion 
limit achieving codes over each level, expansion coding scheme readily achieves the following 
result. 


Theorem 9. For an exponential source, expansion coding achieves the rate distortion pair given 
by 


Ri 

Di 


L 2 

l=-Li 

L 2 

2'di+ 2-^2+i/y2 

l=-Li 


(24) 


(25) 
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Fig. 9. Illustration of successive encoding and decoding. Encoding and decoding start from the highest level. A lower level 
is modeled as one-side distortion (test channel is Z-channel) if and only if estimates in all higher levels are decoded as equal 
to the source. In this illustration, red arrows represent for decoded as equal, while blue ones represent for decoded as unequal. 


for any Li, L 2 > 0, and di G [0, 0.5] for I G {—Li, • • ■ , L 2 }, where pi is given by ( [22| ). 


Proof: See Appendix 


Note that, the last two terms in (251 are a result of the truneation and vanish in the limit 


of large number of levels. In later parts of this seetion, we eharaeterize the number of levels 
required in order to bound the resulting distortion within a eonstant gap. 

2) Successive encoding and decoding: Note that it is not neeessary to make sure xi > xi 
for every I to guarantee x > x. To this end, we introduee sueeessive eoding seheme, where 
eneoding and deeoding start from the highest level L 2 to the lowest. At a eertain level, if all 
higher levels are eneoded as equal to the souree, then we must model this level as binary 
souree eoding with the one-sided distortion. Otherwise, we formulate this level as binary souree 
eoding with the symmetrie distortion (see Figure for an illustration of this sueeessive eoding 
seheme). In partieular, for the later ease, the distortion of eoneem is Hamming distortion, i.e. 
du{xi,xi) = Denoting the equivalent distortion at level I as di, i.e. E,[Xi — Xi] = di. 
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then the symmetrie test ehannel from X; to X; is modeled as 
Pr{Xi = l|Xz = 0} = Pt{Xi = 0|Xz = 1} = 


di 


1 — 2pi + 2di 


Henee, the aehievable rate at level I is given by 

Rx,i = H{pi) - H 


di 


l-2pi + 2di 

Based on these, we have the following aehievable result: 


(26) 


Theorem 10. For an exponential source, applying successive coding, expansion coding achieves 
the rate distortion pairs 

L 2 

R 2 = [oiiRz,i + (! — ««) Rx,i], (27) 

l=-Li 

L2 

D2= (28) 

l=-Li 

for any Li,L2 > 0, and di G [0, 0.5] for I G {—Li, ■ ■ ■ , L 2 }. Here, pi is given by ( |22l ), and the 
values of ai are determined by: 

• For I = L 2 , 


OIIj2 1 1 


• For I < L 2 , 


^2 

ai= n (1 - dk). 

k=l-\-l 


Proof: See Appendix ■ 

In this sense, the aehievable pairs in both theorems are given by optimization problems over 
a set of parameters [d-h-, ■ ■ ■ idif^- However, the problems are not eonvex, so an effeetive 
theoretieal analysis may not be performed here for the optimal solution. But, by a heuristie ehoiee 
of di, we ean still get a good performanee. Inspired by the faet that the optimal seheme models 
noise as exponential with parameter 1/D m the test ehannel, we design di as the expansion 
parameter from this distribution, i.e., we eonsider 


di = 


1 

l + e2V^- 


(29) 


We note that higher levels get higher priority and lower distortion with this ehoiee, whieh 
is eonsistent with the intuition. This ehoiee of di may not guarantee any optimality, although 
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simulation results imply that this can be an approximately optimal solution. In the following, 
we show that the proposed expansion coding scheme achieves within a constant gap to the rate 
distortion function (at each distortion value). 


Theorem 11. For any D G [0,1/A], there exists a constant c > 0 (independent of A and D), 
such that if 


• Li>- log D; 

• L 2 > — log X^D, 


then, with a choice ofdi as in ( [291 ), the achievable rates obtained from expansion coding schemes 
are both within c = 5log{e) bits gap to Shannon rate distortion function, i.e.. 


Ri — Ri^Di) < c, 
R 2 ~ R{,D 2 ) ^ c. 


Proof: See Appendix 


Remark 12. We remark that the requirement for highest level is much more restricted than 
channel coding case. The reason is that number of levels should be large enough to approximate 
both rate and distortion. From the proof in Appendix it is evident that L 2 > — log A is enough 
to bound the rate, however, another — log XD is required to approximate the distortion closely. 
(If only the relative distortion is considered, these extra levels may not be essential.) 


Remark 13. Expansion source coding can be also applied to other source statistics. For instance, 
for Laplacian (two-sided symmetric exponential) sources, the proposed coding scheme can still 


approximately approach to the Shannon rate distortion limit with a small constant gap IpS], 
where the sign bit of Laplacian is considered separately and encoded perfectly with 1 bit, and 
each expanded level is encoded with Hamming distortion, for low distortion regime. 


D. Numerical Results 


Numerical results showing achievable rates along with the rate distortion limit are given in 


Fig. 10 It is evident that both forms of expansion coding perform within a constant gap of the 


limit over the whole distortion region, which outperforms existing scalar quantization technique. 
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Fig. 10. Achievable rate distortion pairs using expansion coding for exponential distribution (A = 1) with one-sided 
error distortion. R{D) (red-solid) is rate distortion limit; (purple) is given by Theorem]^ {D2,R2) (blue) is given 

by Theorem [T^ Linear and non-linear scalar quantization methods are simulated for comparison. 


especially in the low distortion regime. Since samples are independent, the simulations for vector 
quantization are expected to be close to scalar quantization and omitted in this result. 


Theorem 11 shows that this gap is bounded by a constant. Here, numerical results show that 
the gap is not necessarily as wide as predicted by the analysis. Especially for the low distortion 
region, the gap is numerically found to correspond to 0.24 bits and 0.43 bits for each coding 
scheme respectively. 


V. Conclusion 

In this paper, the method of expansion coding is proposed to construct good codes for analog 
channel coding and source coding. With a perfect or approximate decomposition of channel 
noise and original sources, we consider coding over independent parallel representations, thus 
providing a foundation for reducing the original problems to a set of parallel simpler subproblems. 
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In particular, via expansion channel coding, we consider coding over g-ary channels for each 
expansion level. This approximation of the original channel together with capacity achieving 
codes for each level (to reliably transmit messages over each channel constructed) and Gallager’s 
method (to achieve desired communication rates for each channel) allow for constructing near¬ 
capacity achieving codes for the original channel. Similarly, we utilize expansion source coding 
to adopt discrete source codes that achieve the rate distortion limit on each level after expansion, 
and design codes achieving near-optimal performance. Theoretical analysis and numerical results 
are provided to detail performance guarantees of the proposed expansion coding scheme. 

One significant benefit from expansion coding is coding complexity. As indicated in theoretical 
analysis, approximately —21oge -f log SNR and —2log{\^D) number of levels are sufficient 
for the channel coding and source coding schemes respectively. Thus, by choosing “good” 
low complexity optimal codes within each level (such as polar codes [14|, pT|), the overall 
complexity of the coding scheme can be made small for the original continuous-valued channel 
coding and source coding problems. 

Although the discussion in this paper focuses on AEN channels as well as exponential sources, 
expansion coding scheme is a more general framework and its applications are not limited to such 
scenarios. Towards this end, any channel noise and any source with decomposable distribution 
could fit into the range of expansion coding. Moreover, the idea of expansion could also be 
generalized to network information theory, where it can play a role similar to deterministic 
models p5| . The expanded channels are not completely deterministic in our case; they possess 
different noise levels, which may enable the construction of precise models for network analysis. 
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Appendix A 

Proof of Lemma [2] 


The “if” part follows by extending the one given in [361, whieh eonsiders the expansion of a 
truneated exponential random variable. We show the result by ealeulating the moment generating 
function of B. Using the assumption that are mutually independent, we have 


Meit) =E [e‘^] = JJ 

l=—oo 

Noting that B^ are Bernoulli random variables, we have 

.,12' / 


J2‘Bi 


E 


1 + e 


A2' 


+ 1 


1 + e 


A2' 


1 + 


1 + e 


-X2‘ ■ 


Then, using the fact that for any constant a G M, 


no 

1=0 

we can obtain the following for t < X, 


,= 


1 — e 


2"+lc 


1 — 


He 

1=0 


lim I I 

n .—A X 


1=0 


1 + 

1 + 


1 — e 


-A 


1 — e' 


t—X * 


(30) 
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Similarly, for the negative part, we have 


-1 


n(i+ 


) = 


1 - e" 


l=—n 


1 — e 


a2- 


whieh further implies that 


n E 




1 _ gt A X — e 


= lim 


-A2- 


A(1 — e* 


l=—oo 


n^oo 1 — " X — e~^ (A — f)(l — 6“'*^) 


(31) 


Thus, finally for any t < X, eombining equations ( [30| ) and pT] ), we get 

A 


Meit) = 


X — t 


The observation that this is the moment generation funetion for an exponentially distributed 
random variable with parameter A eoneludes the proof. 

The independenee relationships between levels in “only if” part ean be simply verified using 
memoryless property of the exponential distribution. Here, we just need to show the parameter 
for Bernoulli random variable at eaeh level. Observe that for any I G Z, 

Pr{Bz = 1} = Pr{B e UkeN+[^\2k - l),2\2k))} = ^ Pr{2'(2fc - 1) < B < 2'(2A;)}. (32) 

km+ 

Using edf of exponential distribution, we obtain 

Pr{2'(2fc - 1) < B < 2\2k)} = _ g-A 2 '( 2 fc) ^ ^-X 2 ‘(, 2 k) j'gA 2 ' _ _ 


Using this in (|32|), we have 


Pr{Bz = l} = ^e-"2'(2fc) (e"2'-x) = 


k=l 


3^2* + X' 


Appendix B 
Proof of Lemma [5] 


From 0, and r] = log Ez, we have 


qi = 


1 + e2V'Ez X + e2* 


By definition of entropy, we obtain 

H{qi) = -qi loggz - (1 - qi) log(l - qi) 


1 + e 


^ log ^ 


2*-’7 


1 + e2'-’’ 1 + e 


3^ log 


2*-’7 


1 + e- 


2*-') ' 
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When I < T], we obtain a lower bound as 
= 


1 + e' 


^ z^rlog + 6^' ^ 


2l-ri 
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1 + e 


377 log 


2l-v 


= log (1 + e 

(a) 


2l-v 


2l-v 


1 + 

> log(l + 1) — log e • 2’‘~^ 
= 1 -loge-2'-^ 


zrloge ■ 2 


l-rj 


where (a) is due to > 1 and 






On the other hand, when I > rj, we have 


H(m) = 
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1 + e' 


— log 1 + e' 


2'-’) 


2l-ri 
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(fe) 
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1 + 62'-" 


log ^2e^‘ ” j + log + e 


1 + e^‘-^ 

(c) 1 + 2'“^ ■ log e _ 2 i-r, 
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1 + 2*“^ ■ log e 
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(d) 
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< 2’?-^+^ ■loge + 2^-^ -loge 
= 31oge-2^-', 


■ loge 


log e 


1 + 1 + 2 ^-’? + 22 («-’ 7)/2 1 + 2 ^-’? 


1 + e 


2i-V7 


32*-’' 


7377 log 


1 + e 


2*-'' 


p2*-’< 


where 

{b) is from 1 < and ’'/(I + e‘^‘ < 1; 

(c) is from log(l + a) < a loge for any 0 < a < 1; 

(d) is from e" > 1 + a + a‘^/2 > 1 + a for any a > 0; 

(e) is from 1 + 2'"^ ■ loge < (2 + 2'"^ + 22('-’?)/2)(2^-'+i ■ loge) and 1 < (1 + 2^-^)2^-^ for 
any I and r]. 


Appendix C 
Proof of Lemma [3 

By definition, qi = ci® qi, so its behavior is closely related to carries. Note that for any I, we 
have 


qi = q(1 - qi) + qi{l - Ci) = qi + q(1 - 2qi) > qi, 
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where the last inequality holds due to < 1/2 and q > 0. Then, for / < 77 , we have 

H{qi)>H{qi)>l-\oge-2^-\ 

where the first inequality holds due to monotonieity of entropy, and the last inequality is due to 
( [T4l ) in Lemma 1^ For the I > rj part, we need to eharaeterize earries first. We have the following 
assertion: 


Q < 2^-'+^ - 


for I > ? 7 , 


(33) 


1 + ^ 

and the proof is based on the following induetion analysis. For / = 77 + 1, this is simply true. 


beeause q < 1/2 for any 1. Assume ( [3^ is true for level I > rj, then at the I + 1 level, we have 

Q+i = piqi{l - Cl) +Pi{l- qi)ci + (1 - Pi)qiCi + piqiCi 

= Pi{ci + qi- 2qici) + g/Q 
(«) 1 

< 2^ci + qi- 2qiCi) + g/Q 


< - ( 2'^"^+^ 


(c) 


+ 


< 


2 -g-r)+l)+l _ 


l + e2'-’' l + e^'-" 
2 


1 + e- 


2i-»)+i ’ 


where 


(а) is due to pi < 1/2 and Q + g; - 2 g;Q = q (1 - 2qi) + g; > 0 ; 

( б ) is due to the assumption ( |3^ for level Z; 

(c) is due to the faet that 1/[2(1 + > 2/(1 + holds for any Z > 77 . To this end. 


the assertion also holds for level Z + 1, and this eompletes the proof of (33). 


Using ( [3^ , we obtain that for any Z > 77 

qi = qi + q(i - 2gi) 

1 


< 


l + e 

_ c2p—i+i 
< 2^-'+\ 


2i-77 


2'n 


— /+! 


1 + 2’'"^+^ 


l + e 


2 '-') 


+ 


1 + 62'-" 
4 


1 - 


l + e 


2i-V7 


(l + e2'-’’)2 


( 34 ) 
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where the last inequality holds due to (1 + 2^ ’’) > 4 for any I > rj. 

Finally, we obtain 

H{qi) < 

= log(2'^-'+^) - (1 - 2^-'+^) log(l - 2'^-'+^) 

< 2 ^-^+^ + (1 - 2’?-'+^)2’?-^+^21oge 
(f) 

< (/-//)■ 2^-'+^ + (/ - r/)2^-'+^21oge 

< 3\oge- {I -T]) -2^-^+^ 

= 6\og e ■ {I — rj) ■ 2'^~\ 

where 

(d) is from (|3^ and the monotonieity of entropy; 

(e) is from — log(l — a) < 2a loge for any a < 1/2; 

(/) is from 1 — < I — rj for any I > rj. 

From the proof, the information we used for pi is that p; < 1/2, so this bound holds uniformly 
for any SNR. 


Appendix D 
Proof of Theorem [7] 

We first prove that i ?2 aehieves eapaeity. Denote ^ = logF^x and 7] = logF^z- Then, we have 
an important observation that 

1 


Pi = 


1 + e2V2« 


Ql+rj—^i 


(35) 
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which shows that channel input is a shifted version of noise with respect to expansion levels 
(see Fig. |^for intuition). Based on this, we have 


where 


L 2 

R 2 = [H{pi o qi) - H{qi)] 

l=-Li 
(“) ^ 

> 5 ; lH{p,)-H(q,)] 

l=-Li 


{b) 


L 2 


Y, - i/(9)| 

l=-Li 

i2+f? —5 L 2 

Y - E 

l=-Li 

— Li — 1 L 2 

Y - E 

— Li — 1 L 2 

> Y [ 1 -2'“^loge] - Y 2^-'31oge 

/=— Li + 77 —^ 1 = L 2 -\- t )—^-\-1 

(d) 

^ ~ v) ~ log e — log e 


(c) 


(e) 




> log I — — e log e — 3e log ee 

\EzJ 

if) ( Ex\ Ez 

> log I 1 + — 1 - — log e - 4e log e 

( 9 ) / Ex\ 

> log f 1 + — j - 5eloge, 


(36) 


(а) is due to pi®qi= pi{l — g) + (1 — pi)qi > pi, and monotonicity of entropy; 

(б) follows from ( [35] ); 

(c) follows from ( [T3] ) and ( [T?] ) in Lemma 

(d) holds as 

—L\ — \ —L\ — \ 

Y 2^"’' < Y 2'"'^ = 2-^^-^, 

l=-Li+ri-i 


l=—oo 


and 


L2+ri-^+l L2+ri-^+l 

Y^ 2^"^ < Y, 2^'"' = 

1 = L2 1=00 
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(e) is due to the assumptions that Li > — log e — rj, and L 2 > — loge + ^; 

(/) is due to the faet that 

log (1 + f) - log (^) = log (1 + f) < f logo. 

as log(l + a) < a loge for any a > 0; 

(g) is due to the assumption that SNR = Ex/Ez > 1/e. 

Next, we show the result for Ri. Observe that 

L2 

Ri=Y. [H{pi®qi)-H{q,)] 

l=-Li 

l=-Li 

L2 L2 

= E \H(p, 0 «) - + x; \H(q,) - H(q,)] 

l=-Li 

L2 

= 4 _ [H(q,) - H(q,)] 


l=-L, 


(i) 


l=-Li 

V 


L 2 


> R 2 — ^ [1 — (1 — 2^ ^loge)] — ^ [6{l — g)2 ’‘~^'^\oge — O] 


-L 

V 


r/+l 


^2 


= R 2 -Y^ 2 '-^ log e - ^ 6(/ - r/)2-'+'? log« 

-Li ri+l 

(j) . 

> /?2 — 2 log e — 12 log e 


(fc) 


E^ 


> log I 1 + — ) — 5e log e — 14 log e, 


where 


{h) is due to qi > qi, whieh further implies pi® qi > pi ® qp, 

(i) follows from ( [T4| ) and ( [T5] ), together with the faet that H{qi) < 1 and H{qi) > 0 for any /; 

(j) follows from the observations that 


^ 2'-^loge<loge-2-^- ^2' = 21oge, 


l=-L^ 


l=—oo 


L 2 


6(/ — p)2 log e < 6 log e • ^ {I — g) -2 = 12 log e; 

l=ri+l i=»?+l 


and 
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{k) is due to 

Thus, choosing c = 19 log e completes the proof. Note that, in the course of providing these 
upper bounds, the actual gap might be enlarged. The actual value of the gap is much smaller 
(e.g., as shown in Fig. numerical result for the capacity gap is around 1.72 bits). 


Appendix E 
Proof of Lemma [8] 


Note that the maximum entropy theorem implies that the distribution maximizing differential 
entropy over all probability densities / on support set IR+ satisfying 

poo poo 

/ f{x)dx = 1, / f{x)xdx = 1/A, 

Jo Jo 

is exponential distribution with parameter A. Based on this result, in order to satisfy E[(i(X, X)] < 
D, where d(X,X) = oo for X < X, we have to restrict X > X with probability 1. To this end, 
we have 


/(X;X) = h{X) - h{X\X) 

= log(e/A) - h{X - X|X) 

> log(e/A) - h{X - X) 

> log(e/A) — log(eE[X — X]) 

> log(e/A) — log(eiA) 

= -log(AZl). 


To achieve this bound, we need X — X to be exponentially distributed and independent with X 
as well. Accordingly, we can consider a test channel from X to X with additive noise Z = X — X 
distributed as exponential with parameter l/D, which gives the conditional distribution given 


by (21). 


Appendix F 
Proof of Theorem [9] 

Due to decomposability of exponential distribution, the levels after expansion are independent, 
hence, the achievable rate in this theorem is obtained by additions of individual rates. On the 
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Other hand, for the calculation of distortion, we have 

Di = E[X - X] 


= E 


^2 


2'X; - 2% 


l=—oo 


l=-L 

oo 


L2 00 —Li — 1 

(“) o/j , 0 / 


2^di + 2^pi + J 2 

=— 1 j \ /=Z/2H“1 I ——00 

L2 00 —Li — 1 

< 2 2^di+ 2-'+V^^+ Y 2 

l=—Li 1=L2+1 l=—oo 

L 2 


(b) 


1-1 


(c) 


Y + 2"^2+i/^2 ^ 


l=-Li 


where 

(а) follows from pi — pi = dp 

(б) follows from 

1 

Pi = -< 


< 


= 2"^'+V'^^ 


1 + ~ 1 + (1 + A2' + A22272) - A22272 

and pz < 1/2 for any 1; 

(c) follows from 

OO —Li —1 

Y 2-^ = 2-^\ and Y 2'= 2"^h 

/ = L 2 + 1 l = — oo 

Appendix G 
Proof of Theorem [T^ 

By the design of coding scheme, if all higher levels are decoded as equivalence, then they 
must be encoded with one-sided distortion. Recall that for Z-channel, we have 


Pr{X;^X,}=Pr{X, = 1,X, = 0} = d,. 


Hence, due to independence of expanded levels, 

L 2 

ai= n ~ 

k=l+l 

Then, at each level, the achievable rate is i?z,z with probability ai and is i?x,z otherwise. From 
this, we obtain the expression of R 2 given by the theorem. On the other hand, since in both 
cases we have pi — pi = di, the form of distortion remains the same. 
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Appendix H 
Proof of Theorem [TT] 

Denote 7 = — log A, and ^ = — logD. Then, from D < 1/A, we have 

7 + ^ > 0 . 


( 37 ) 


By noting that pi and di are both expanded parameters from exponential distribution, we have 

1 1 

Pi = 


d,= 


1 + e^^'' 

1 


1 + ^ 
1 


1 + e2Vo 1 + e2'+« ■ 

Henee, pi is shifted version of di (analog to the ehannel eoding case), i.e., 

di = pi+^+^. 


Using this relationship, we obtain 

IHipi) - H{d,)] s ^ H{pi) - ^ ff(B+,+e) 

l=—Li l=—L\ l=—Li 

L2 -I' 2 + 7 +? 

= E H(p,) - Y. 

l=—Li Li+7 +^ 

-Li+7+^-1 L2+7+C 

= E E 

l=—Li /=L2+1 

(c) 

<7 + '^, 

where 


(a) follows from ( [38] ); 

(b) follows from ( [37] ) and theorem assumptions; 

(c) follows from 0 < H{pi) < 1 for any 1. 

From the expression of Ri, we have 

L 2 


Ri= H{pi)-{l-pi + di)H 

l=-Li - 

= E + E 

l=-Li l=-Li 

L 2 


di 


< 7 




«=-Li >- 


H{di)-{l-pi + di)H 


1 - Pi+ di 
H{di)-{l-pi + di)H 
di 


di 


1 - Pi+ di 


1 - Pi+ di 


(38) 


(39) 


(40) 
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where ( |39| ) is used to obtain the last inequality. It remains to bound 
A, A H(d,) - (1 - p, + 

\1 - Pi+ dij 

= (1 - Pi) log(l - Pi) - (1 - di) log(l - di) - {l-pi + di) log(l -pi + di). 
For this, two oases are eonsidered: 

1) For I < —di and pi are elose and both tend to 0.5. More preoisely, we have 

id) 

A; < -{l-pi + di) log(l -Pi + di) 


(e) 

< 2{pi -di) log e 


if) 

< 2 


- - 
2 


loge 


= 2'+« loge, 


(41) 


where 

(d) follows from the faet that (1 — a) log(l — a) is a deoreasing funotion over [0,0.5], 
henee, (1 - pi) log(l - pi) < (1 - di) log(l - di); 

(e) follows from the observation that —(1— a) log(l — a) < 21oge-Q! for any a G [0,0.5]; 
(/) follows from the faet that pi < 0.5 and 

j _ ^ \ ^ 9^+?—1 

' 1 + - 1 + (1 + 2 ■ 2 '+«) - 2 

where the first inequality is due to e" < 1 + 2a for any a G [0,1] (2*+^ < 1 due to 
I < —0, and the last inequality holds for any 1. 

2) On the other hand, for I > —di tends to 0, so as 1 — pi and 1—pi + di get elose. More 
preeisely, we have 

(9) 

A; < -(1 - di) log(l - di) 

(h) 

< 2di log e 
(0 

< 2-'-« log e, (42) 

where 

{g) follows from the faet (1 - pi) log(l - pi) < {I - pi + di) log(l - pi + di)\ 

{h) follows from the observation that —(1 — a) log(l — a) < 2a log e for any a G [0, 0.5]; 
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(z) follows from the fact that 

rj — ^ ^ ^ ^ _ 9—1—^—1 

1 + e2'+' - e2'+' - 2 ■ 2'+« 

where the second inequality holds from e" > 2a for any a > 1 (2^+^ > 1 due to 
I > -0. 

Putting ( |4T| ) and ( |4^ back to ( |4^ , we have 

-S L2 

-Ri < 7 + ^ + log e ■ ^ 2 '+^ + log e • ^ 2~'-~^ 

l=-Li l=-^+l 

< 7 + ^ + 2 log e + log e 


= R{D) + 31oge, 

where we use the definitions of 7 and such that 7 + .^ = R{D). 
Finally, using the result from Theorem that 


^2 


D <Di< 2 U 1 + 2-^2+V^^ + <D + 2-^2+i/^2 ^ 2 - 

l=-Li 


we obtain 


0) lose 

RiD) < R{D,) + ^iD,-D) 


loge^^_^2+l / \2 I 


(k) 

< i?(Di) + ^(2-^2+VA'' + 2 
(0 

< R{Di) + 2.5 log e, 


where 


(43) 


(44) 


(45) 


(j) follows from R{D) is convex such that for any f3 and a, 

R(/9) > R(a) + R'(a)(/)-a), 

where R!{a) = — log e/a is the derivative, and setting a = D, [3 = Di completes the proof 
of this step; 


{k) follows from ( |44l ); 

(Z) follows from theorem assumptions that Li > —\ogD and L 2 > — logA^D. 


Combining (45) with (43), we have 


Ri < R{Di) + 5.5 loge, 
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which completes the proof for Ri and Di by taking c = 5.5 log e. 
For the other part of the theorem, observe that 


H 


dj 


l-2pi + 2di 


>{l-pi + di)H 


di 


1 - Pi+ di 


Hence, for any —Li < I < Li, we have Rx,i < Rz,i- Thus, we have R2 < 
the observation that Di = D 2 , we have R 2 < R{D 2 ) + 5.5 log e. 


Ri. Combing with 



